Skip to content

feat(agent): add custom_validator hook with retry to AsyncAgent.query#77

Merged
anand-simbian merged 2 commits into
mainfrom
feat/async-query-custom-validator
Jun 8, 2026
Merged

feat(agent): add custom_validator hook with retry to AsyncAgent.query#77
anand-simbian merged 2 commits into
mainfrom
feat/async-query-custom-validator

Conversation

@anand-simbian

Copy link
Copy Markdown
Contributor

Summary

  • Adds optional custom_validator + max_validation_retries parameters to AsyncAgent.query. Validator (sync or async) inspects the LLM result; returning a feedback string triggers a re-prompt that includes the feedback so the LLM can self-correct. Bounded by max_validation_retries (default 2 extra calls).
  • Useful for callers that need a structural / semantic check beyond what strict_json's schema enforcement provides — e.g. validating that every UUID the LLM emits belongs to a known input set.
  • Defaults preserve existing behavior (no validator → single call, same as today).
  • Bumps version 3.4.3 → 3.4.4.

Motivation

Consumer use case: SimbianOS filter-observables step where the LLM occasionally garbles a hypothesis_result_id UUID by a single character, crashing downstream DAO lookups. With this hook the caller can pass a UUID-in-input-set validator and TaskGen handles the retry/feedback loop in one place instead of every caller wrapping query in their own retry helper. Tracking comment: https://github.com/simbianai/SimbianOS/pull/4702#discussion_r3288175018

Test plan

  • Unit-test AsyncAgent.query with a validator that returns None (no retry, single call)
  • Unit-test with a validator that rejects once then accepts (one retry, feedback appended to prompt)
  • Unit-test with a validator that always rejects (returns last result after max_validation_retries)
  • Smoke-test consumer integration in SimbianOS PR #4702

Adds an optional `custom_validator` callable (and `max_validation_retries`)
parameter to `AsyncAgent.query`. When supplied, the validator is invoked
with each LLM result; if it returns a non-empty feedback string, the
prompt is re-issued with the feedback appended so the LLM can self-correct.
Bounded by `max_validation_retries` (default 2 extra calls).

This is useful for callers that need a structural / semantic check that
strict_json's schema enforcement can't express — for example, validating
that every UUID the LLM emits belongs to a known input set.

Defaults preserve existing behavior (no validator, single call).

Bumps version 3.4.3 -> 3.4.4.
Comment thread taskgen/agent.py Outdated
Comment thread pyproject.toml Outdated
@anand-simbian anand-simbian merged commit e200480 into main Jun 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants